Benchmarking computers and computer networks

The benchmarking concept is not new in the field of computing or computer networking. With “benchmarking tools”, one usually refers to a program or set of programs, used to evaluate the performance of a solution under certain reference conditions, relative to the performance of another solution. Since the 1970s, benchmarking techniques have been used to measure the performance of computers and computer networks. Benchmarking of applications and virtual machines in an Infrastructure-as-a-Service (IaaS) context is being researched in a BonFIRE experiment and the benchmarking of wired and wireless computer networks is put forward as a research topic in the research projects CREW and OneLab2. In this paper, we elaborate on the interpretation of the term “benchmarking” in these projects, and answer why research on benchmarking is still relevant today. After presenting a high-level generic benchmarking architecture, the possibilities of benchmarking are illustrated through two examples: benchmarking cloud services and benchmarking cognitive radio solutions.